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As the United States continued its shift from a manufacturing- to a service-based 
economy in the late 1980s and early 1990s, researchers reported that changes in 
employment patterns would require workers to have better communication skills and to 
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be both literate and proficient in English (McGroarty & Scott, 1993). Not surprisingly, 
there was a rise in the number of workplace education programs for both native and 
non-native speakers of English. The U.S. Department of Education's National 
Workplace Literacy Program (NWLP), which funded demonstration workplace projects 
offering instruction in basic skills, literacy, and English as a Second Language (ESL), 
fueled this increase by funding more than 300 projects between 1988 and 1994. 
Forty-nine percent of these projects included at least some ESL instruction. 

With this increase in workplace instructional programs, a need has arisen for 
procedures to evaluate program effectiveness. Evaluations of ESL workplace programs 
seek to determine if the attention given to improving basic skills and English language 
proficiency has made a change in the participant and in the workplace. They also 
identify practices associated with program effectiveness so that successes can be 
replicated (Alamprese, 1994). This digest examines evaluation measures and activities 
used in workplace programs, and discusses issues associated with the evaluation of 
workplace ESL programs. 

EVALUATION MEASURES AND ACTIVITIES 

Because numbers alone cannot show the depth or the breadth of a program's impact, 
evaluations often use both quantitative and qualitative measures to gauge success in 
attaining program outcomes (Padak & Padak, 1991). Qualitative measures include 
focus groups and individual interviews, workplace observations, and portfolios of learner 
classwork (Alamprese, 1994). Quantitative measures include commercially available 
tests, scaled performance ratings, and some program-developed assessment tools, 
such as portfolios. 



Focus Groups and Stakeholder Interviews 

What is examined in an evaluation is determined by stakeholders' (employers, labor 
unions, participants, teachers, funders) stated goals, expected outcomes for the 
program, and the resources available to the evaluator (Patton, 1987). As stakeholders 
may have different, possibly conflicting goals, it is important to clarify these goals and 
achieve a consensus beforehand as to which goals are most important to examine with 
the available resources (Fitz-Gibbon & Morris, 1987). The Information gathered from the 
focus groups and stakeholder interviews should be recorded and accessible to the 
program and to the evaluators throughout the program. 



Qbservations 

Task analyses are generally used in curriculum development as educators observe and 
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record their observations of the discrete steps included in workplace tasks such as 
setting up the salad bar for a cafeteria or making change for a customer at the cash 
register. The recorded observations are then plotted on a matrix of basic skills or 
English language skills. Although programs have relied on these analyses as a key data 
source for workplace outcomes (Alamprese, 1994), they do not represent the totality of 
skills used at the workplace. In order to better understand the "range" of skills needed 
for workplace success, other workplace-related activities such as staff meetings and 
union functions should also be observed. 



Participant and Supervisor Interviews 

Pre-program interviews with participants solicit information on their goals, their reasons 
for enrolling in the classes, and their perceived basic skills and English language needs 
for the workplace. When matched with exit interview data, these data provide 
information to evaluate program outcomes. Because the purpose of these interviews is 
to obtain information about learner perceptions rather than to assess learner skills, it is 
advisable to use the native language when interviewing participants with low English 
skills. 

Similarly, the direct supervisors of participants should be interviewed both before and 
after the program to compare initial assessment of learner needs and expected 
outcomes with actual results. It is also useful to interview the direct supervisors midway 
through the program for their feedback on worker improvement and to identify unmet 
needs. 



Tests and Other Types of Assessment 

Commercially available tests are commonly used as sources of quantitative data. The 
perceived objectivity of these tests and their long tradition of use make them appealing 
to managers and funders who often use them to make decisions regarding the 
continuation of a program. And, in fact, test-taking is a skill all learners need, and it is 
likely that ESL participants will come across this type of test in other contexts, as well. 

Two commercially available tests that include workplace- related items and are often 
used in ESL programs are the Basic English Skills Test (BEST) and the Comprehensive 
Adult Student Assessment System (CASAS) ESL Appraisal. These instruments are 
easy to use, their reliability has been tested, and they allow for comparison among 
programs. The objections to these tests are that they may not measure what has been 
taught in the classroom, and they may have little applicability to specific workplace tasks 
or to a particular workplace. And, as with all tests, when interpreting results, evaluators 
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and program staff should be aware that some errors may be due to ESL participants' 
unfamiliarity with the format of the tests rather than to lack of content knowledge. 

Because of the limitations of commercially available tests, a complete evaluation of 
learner progress requires using tests created for the program. "Performance-developed 
tests" are designed to measure the learner's ability to apply what has been learned to 
specific workplace tasks (Alamprese & Kay, 1993). Because these tests are developed 
from authentic materials (e.g., job schedules, pay stubs, and union contracts) from 
participants' own workplaces, the content is appropriate and likely to be familiar to the 
participants. 

Another assessment measure is the "portfolio" of learner work. Portfolios often include 
samples of class work, checklists where learners rate their progress in basic and 
workplace skills, and journals where they record their reactions to class and workplace 
activities. Like interviews, these measures can provide vital information on learner 
attitudes and concerns. They are also a venue for self-assessment, and allow 
participants who are unable or unwilling to express themselves orally, or who have 
difficulty with formal tests, to demonstrate progress towards their goals. 



Quantifying Qualitative Measures 

To increase credibility and help ensure reliability of qualitative measures, evaluators 
collect multiple types of evidence (such as interviews and observations) from various 
stakeholders around a single outcome (Alamprese, 1994; Patton, 1987; Lynch 1990). 
Data collected from the various measures can then be arranged into matrices. This 
chart-like format organizes material thematically and enables an analysis of data across 
respondents by themes (see Fitz-Gibbon & Morris, 1987; Lynch, 1990; and Sperazi & 
Jurmo, 1994). 

Questionnaire and interview data can be quantified by creating a scale that categorizes 
responses and assigns them a numeric value. Improvement in such subjective areas as 
worker attitudes can then be demonstrated to funders and managers in a numeric or 
graphic form. 

ISSUES IN PROGRAM EVALUATION 

Many issues surround program evaluation for workplace ESL instruction. Stakeholders 
may have unrealistic expectations of how much improvement a few hours of instruction 
can effect. It is unlikely that a workplace ESL class of 40-60 hours will turn participants 
with low-level English skills into fluent speakers of English. Therefore, when interpreting 
findings, it is important for stakeholders to realize that ESL workplace programs may not 
provide enough practice time to accomplish substantial progress in English language 
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proficiency. 

The measurement of workplace improvement presents a special challenge, especially in 
workplace programs at hospitals, residential centers, and restaurants. What measures 
of workplace productivity exist where there is no product being manufactured? Improved 
safety (decreased accidents on the job) is a quantifiable measure, as is a reduction in 
the amount of food wasted in preparation. But how is improved worker attitude 
measured? Some ESL programs measure success by counting the increased number 
of verbal and written suggestions offered on the job by language minority workers or by 
their willingness to indicate lack of comprehension on the job (Mikulecky & Lloyd, 1994; 
Mrowicki & Conrath, 1994). Other programs record participant requests to be 
cross-trained or to learn other jobs at their workplaces (Alamprese & Kay, 1993). A 
long-term view is often needed, however, to discern changes in worker performance 
and in workplace productivity; longitudinal studies, where stakeholders are interviewed 
six months to a year after completion of a program, are recommended. 

Even if data from longitudinal studies is available, it is not accurate to place all credit for 
improvement in worker attitude or workplace productivity (or blame for lack thereof) on 
the instructional program. Sarmiento (1993) asserts that other factors (Are there 
opportunities for workers to advance? Are the skills of all workers appreciated and 
used? Is worker input in decision making valued?) need to be considered when 
evaluating workplace programs. However, for ESL participants who come from cultures 
where assertiveness, ambition, and speaking up on the job may not be valued, the 
presentation of opportunities to succeed is not enough. Advancing oneself at the U.S. 
workplace is a cross-cultural skill, which, like language and literacy skills, must be 
taught. 

Finally, funding is an important issue in evaluation. The activities described above 
(focus groups, interviews in English or in the native language, program-developed 
assessment instruments, extensive contacts with all stakeholders from before the 
program begins until months after completion) are costly. As federal funds are unlikely 
to be available, evaluations need to be structured so that they can provide practical 
information to the employers funding them. 



CONCLUSION 



Evaluation is a complex process that involves all stakeholders and must be an integral 
part of workplace ESL instructional programs before, during, and after the programs 
have been completed. When done appropriately, it can increase program effectiveness 
by providing valuable information about the impact of programs and highlighting areas 
where improvement is needed (Jurmo, 1994). And, a rigorous and complete evaluation 
can identify replicable best practices, enabling a program to serve as a model for other 
workplace ESL instructional programs. 
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